Early therapy response assessment of lesions

ABSTRACT

For therapy response assessment, texture features are input for machine learning a classifier and for using a machine learnt classifier. Rather than or in addition to using formula-based texture features, data driven texture features are derived from training images. Such data driven texture features are independent analysis features, such as features from independent subspace analysis. The texture features may be used to predict the outcome of therapy based on a few number of or even one scan of the patient.

RELATED APPLICATION

The present patent document claims the benefit of the filing date under35 U.S.C. §119(e) of Provisional U.S. Patent Application Ser. No.61/882,143, filed Sep. 25, 2013, the disclosure of which is herebyincorporated by reference.

BACKGROUND

The present embodiments relate to early therapy response assessment. Inparticular, computer-aided response assessment is provided.

The response of a tumor under therapy is measured based on geometricmeasures, such as the diameter (e.g., RECIST or WHO criteria) or volume.A clinician monitors the geometric measures in radiological scans (e.g.CT or MRI) for a long period of time until a decision can be made aboutoutcome. Many rounds of therapy are performed before the clinician makesa decision based on the geometric measures about whether the therapy islikely to succeed or not. Long periods of therapy may result in higherdosage for the patient (e.g., radiation therapy and/or x-ray imaging forthe geometric measures) and greater cost of the therapy drugs.

Typically, the lesion appearance (e.g. enhancement pattern) changesearlier than geometric changes of the lesion occur. The lesion maymaintain its size for a longer time while its tissue is already turningnecrotic. The texture of a lesion gives insight into the therapyresponse at a much earlier stage, when the size or shape of the lesionis still largely unaffected. Using texture as a parameter, it may bepossible to identify cases with a similar medical condition in adatabase for which the applied therapy and its outcome are known. Thismay help the doctor in estimating the effectiveness of differenttherapies and chose the best strategy. However, such approaches are timeconsuming and/or difficult for a doctor or other person to perform.

SUMMARY

Systems, methods, and computer readable media are provided for therapyresponse assessment. Texture features are input for machine learning aclassifier and for using a machine learnt classifier. Rather than or inaddition to using formula-based texture features, data driven texturefeatures are derived from training images. Such data driven texturefeatures are independent analysis features, such as features fromindependent subspace analysis. The texture features may be used topredict the outcome of therapy based on a few number of or even one scanof the patient.

In a first aspect, a method is provided for therapy response assessment.Pre-therapy and post therapy medical images of a patient are obtained.The medical images represent at least one lesion of the patient. Aprocessor convolves the pre-therapy and post therapy medical images witha texture feature learned from training images. The processor classifiesa therapy response of the lesion with a machine-learnt classifier with aresult of the convolving as an input feature to the machine-learntclassifier.

In a second aspect, a non-transitory computer readable storage mediumhas stored therein data representing instructions executable by aprogrammed processor for therapy response assessment. The storage mediumincludes instructions for: with only one or two scans, extractingtexture features for a lesion with a filter kernel, the filter kernelbeing independently based on image data; and predicting an outcome oftherapy on the lesion, the predicting being a function of the texturefeatures.

In a third aspect, a method is provided for therapy response assessment.A processor subjects patches of lesions represented in a plurality oftraining frames of data to independent subspace analysis. The processorcreates a number of image filters from the independent subspace analysisof the patches and calculates texture features by application of theimage filters to the lesions as represented in the training frames ofdata. The processor learns a predictor of therapy response as a functionof the texture features.

Any one or more of the aspects described above may be used alone or incombination. These and other aspects, features and advantages willbecome apparent from the following detailed description of preferredembodiments, which is to be read in connection with the accompanyingdrawings. The present invention is defined by the following claims, andnothing in this section should be taken as a limitation on those claims.Further aspects and advantages of the invention are discussed below inconjunction with the preferred embodiments and may be later claimedindependently or in combination.

BRIEF DESCRIPTION OF THE DRAWINGS

The components and the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of theembodiments. Moreover, in the figures, like reference numerals designatecorresponding parts throughout the different views.

FIG. 1 shows an example process for therapy response assessment usinggeometric measures;

FIG. 2 shows an example process for therapy response assessment usingtexture features;

FIG. 3 is a flow chart diagram of one embodiment of a method fortraining in therapy response assessment;

FIG. 4 illustrates example texture features learned from trainingimages;

FIGS. 5A-C show results of convolving the features of FIG. 4 with threedifferent lesions;

FIG. 6 is a flow chart diagram of one embodiment of a method forapplication in therapy response assessment; and

FIG. 7 is a block diagram of one embodiment of a system for therapyresponse assessment.

DETAILED DESCRIPTION OF EMBODIMENTS

In therapy response assessment, a computer assists a doctor inestimating the effectiveness of different cancer therapies and choosingthe best strategy. Given the variability of lesions, imaging settings,situations, and number of cases, it may take a doctor or other medicalprofessional an unreasonable amount of time to identify similar casesand patients for making an informed prediction. As a result, there maybe a great amount of variability in predictions by different medicalprofessionals. A computer may more quickly access the information andmore consistently predict therapy response. Given the subtlety ofprediction from, at least in part, medical images, a processor maybetter perform or at least provide a more efficient second perspectiveon therapy response assessment.

In order to judge the state of a tumor or lesion in an objective manner,measures of therapy response suitable at early therapy stages are used.The effect of the therapy is assessed as early as possible. Using twoscans, or possibly with only one scan, the predictor may decide whetherthe therapy will fail or succeed. Queues that may be missed by a doctorwith such a few images may be identified by the processor applying thepredictor. The decision making happens at an earlier stage.

FIG. 1 shows an approach using geometric measures where a pre-therapyscan is performed and then a sequence of N (e.g., 5 or more) scans areperformed after therapy or during a sequence of therapy applications.The clinician then uses the scans to determine geometric changes topredict the success or failure of the therapy. Meanwhile, the patienthas been exposed to the medications and/or radiation associated withtherapy and/or monitoring many times. Conversely, FIG. 2 shows acquiringthe pre-therapy scan and a single post therapy scan. Texture featuresreflecting cell density, vasculature, necrosis, hemorrhage, othercharacteristics, or combinations thereof are extracted and used forautomated (e.g., machine learnt) prediction of success or failure of thetherapy. The prediction may be made earlier in the process, possiblyavoiding further therapy where the outcome is likely negative. Theprediction may be made based on the pre-therapy scan before therapy.Early therapy outcome prediction is provided, possibly reducing exposureof a patient to therapy that is likely not successful.

In the prediction, the texture of organs and/or lesions is used topredict the likelihood of future cancer development. The features areextracted using Independent Subspace Analysis (ISA) or other data-drivenapproach. Rather than or in addition to using programmer designedformulas for measuring texture, training scans are used to determinetexture that is appropriate for and/or determinative of outcome for agiven type of lesion, organ, therapy, and/or other situation. This isdifferent from traditional features where a fixed formula describes howto calculate the feature value. With ISA or other data-drivendevelopment of the texture features, different sets of training imagesand ground truth labels may generate different image filters, and hencedifferent feature values. The advantage of such a data-driven approachis that, as the training dataset grows larger, more meaningful patternsof texture may be discovered without relying on expert knowledge tomanually define heuristic and sub-optimal mathematical formulas forpatterns. The learning-based approach also allows training the systemfor different therapy types (e.g. chemo-therapy, targeted therapy,radiation therapy) resulting in therapy-specific response features thatare automatically learnt. Data-driven texture features allow fordifferent features to be learned for different types of lesions,tissues, therapy, and/or imaging modalities.

FIGS. 3 and 6 show methods for therapy response assessment. The methodfor therapy response assessment may be a method to learn how to accessoutcome of therapy or may be a method for assessing the outcome with alearnt predictor. FIG. 3 is directed to machine training of the therapyoutcome predictor. FIG. 6 is directed to application of a machine-learnttherapy outcome predictor. In both cases, a machine, such as aprocessor, computer, or server, implements some or all of the acts. Thesystem of FIG. 7 implements the methods in one embodiment. A user mayselect the image files for application of the therapy response predictorby the processor, select the image from which to learn features by aprocessor, and/or identify a region of interest. Use of the machineallows processing large volumes (e.g., images of many pixels and/or manyimages) of information that may not be efficiently handled by humans (atleast in comparative time frames for a given volume of information), maybe unrealistically handled by humans in the needed time frame, or maynot even be possible by humans due to subtleties and/or timing.

The methods are provided in the orders shown, but other orders may beprovided. For example in FIG. 6, acts 58 and 60 may be performed inparallel or opposite order.

Additional, different or fewer acts may be provided. For example, act 44of FIG. 3 is not provided. As another example, acts 60 and/or 64 of FIG.6 are not provided. In yet other examples, acts for capturing imagesand/or acts using detected information are provided. Acts forconfiguring, input, or output may be provided.

FIG. 3 shows a method for learning in therapy response assessment. Aprocessor performs the learning, such as by performing acts 44-52. Theprocessor both creates texture features from training data in acts 46-48and learns a predictor with the texture features as inputs in act 52.The learnt feature may be used by the processor to calculate a featurevalue and/or by the processor to train a classifier.

In act 40, one or more images of an object are obtained. The images areobtained by data transfer, capture, and/or loading from memory.

The images are frames of data representing a patient at given times. Theimages are scalar values or display values (e.g., red green blue (RGB)).The images may have been previously displayed or have not yet beendisplayed.

The images are acquired from a scan. The images are captured using anyone or more sensors. For example, images of organs are captured usingx-ray, computed tomography (CT), fluoroscopy, angiography, magneticresonance, ultrasound, positron emission tomography, or single photonemission computed tomography. A given pre or post therapy scan providesone or more images. For example, images for different phases of amultiple phase contrast enhanced scan (e.g., native, arterial, andvenous phases of contrast agent wash in and/or wash out) are acquired.As another example, Dual Energy CT images, including an iodine map, areacquired as part of a scan.

The pre and post therapy scans are the same or different types of scans.Each scan provides one or multiple images. Multiple images of the sameor different patients use the same or different imaging modality withthe same or different settings (e.g., field of view).

The object of interest in a medical image may be an organ, a cyst, atumor, calcification or other anomaly. Any type of lesion may be imaged.The organ surrounding the lesion may be scanned. The image may representhealthy tissue. The image may represent the lesion in two orthree-dimensions. For example, the image may be of pixels or voxels.

For training, images are acquired for the same type of lesion with orwithout other common arrangements (e.g., patient characteristics (malevs. female or age), imaging modality, organ, type of therapy, and/orstage). Pre and post therapy images for many, such as tens, hundreds,thousands, or more, patients are obtained. The images are associatedwith or linked to known outcomes. For example, all of the images arelabeled with ground truth. For creating texture features, just theimages associated with positive outcome are used. Alternatively, justimages associated with negative outcome or both positive and negativeoutcomes are used. For training the predictor, images labeled for bothpositive and negative outcome are used.

In act 42, one or more regions of interest (ROI) are identified in eachof the images. Manual (e.g., user input) or automated tumor detection orsegmentation is used to find the region of interest. Any now known orlater developed region of interest identification may be used.

The region of interest is a bounding box or other shape enclosing thelesion or is a segmentation specifically of the lesion. More than oneregion of interest may be provided on a given image. For example,multiple lesions are represented in a given image. As another example,one or more regions of interest for healthy tissue are identified. Inone embodiment, nested regions of interest are provided, such as one foran organ as a whole or in part and a sub-set of that region being aregion for a lesion.

The region of interest is to be used for feature extraction. Tumorassociated features may be extracted from within the region of interest.The approach is to combine multiple sets of different features fortraining a classifier. Such features may also be extracted from otherreference regions in the image data, such as the whole liver parenchyma,to complement the tumor features extracted from within the lesionspecific ROI and to provide the classifier with additional information.

The delineated lesion region of interest may be expanded to assure thatthe border or gradient from lesion to healthy tissue is included. Theregion of interest is dilated (enlarged) by the processor to includesurrounding information. The dilation is by a percentage (e.g., 5-10%expansion), by a number of samples (e.g., adding 2-10 samples from acenter), or by another process. Alternatively or additionally, theregion of interest is originally defined by the user or processor toinclude the surrounding information. In alternative embodiments, noexpansion is provided.

The region of interest may be sub-divided. For example, a moving windowis defined to cover a part of the region of interest. By moving thewindow with any step size (e.g., 1 pixel, 2 pixels, or other distance),overlapping sub-sets of the region of interest are defined. Thesepatches or sub-sets are uniformly defined within the region of interestwith a fixed step size. Any size of patch may be used, such as 8×8pixels. In alternative embodiments, the patch size and/or step size varyor are not uniform.

In act 44, the patches are whitened by the processor. Within each kernelor patch, the values are altered to set the mean to a zero value andmake the variance equal to one. Other normalization may be used. In oneembodiment, the patches are whitened with a principal componentanalysis. Alternatively, whitening is not performed.

In act 46, independent subspace analysis (ISA) or independent componentanalysis (ICA) is applied to the patches, such as the whitened patches.The independent analysis examines the data of the patches to find textpatterns. The processor analyzes many or all of the patches of many orall of the regions of interests from the different patients to learn oneor more texture features. Data-driven machine learning is used by theprocessor to recognize one or more patterns common across variouspatches and patients. Both ISA and ICA find features that formindependent subspaces, but ISA is found to be more robust to localvariations than ICA.

In one embodiment, the patches are subjected to independent subspaceanalysis (ISA) by the processor. In another embodiment, the patches aresubjected to independent component analysis by the processor. Thepatches are decomposed into different components to find patterns. Otherpattern recognition, with or without independence, may be used.

The independent analysis or other pattern recognition by the processorfrom the many examples results in a number, N, of image filters. In act48, the image filters are created by the application of the independentanalysis in act 46. Any number of common patterns in the varioustraining examples may be created. For example, five image filters arelearned from the training images.

To distinguish texture of lesions with desired response from others, theground truth is used. The training data associated with desired outcome,as indicated by the ground truth, is used for creating the imagefilters. Alternatively or additionally, other training data is used. Inother embodiments, the independent analysis learns texture features thatdistinguish between texture with desired outcome from texture with anundesired outcome, so the training data is divided into sets based onthe ground truth. In alternative embodiments, the image filters arecreated without consideration of ground truth. Texture common to bothnecrotizing and non-necrotizing lesions are found by computerimplemented analysis.

The image filters have a same size and shape as the patches.Alternatively, the image filters have a different size and/or shape asthe patches. The image filters are binary masks or include any number ofgradations, such as including an average or full dynamic range of thepatches from the training data.

FIG. 4 shows five example image filters learned by a processor frompattern recognition using independent subspace analysis. The imagefilters were created in act 48 from CT images of liver tumors ofpatients. These image filters represent more common patterns across theset of lesions for the training data. The same, different, or no imagefilters are created for healthy tissue. Given different training data,different image filters may result.

In act 50, texture features are calculated by the processor. The imagefilters (see FIG. 4 for example) represent texture features for a givensituation, but are not themselves the inputs for learning a predictor.The image filters are applied to the training data to calculate valuesfor the texture features. The image filters are applied to the sametraining data from which the image filters were created. Alternatively,a different set of training data is used.

The image filters are applied to the appropriate regions of interest.For image filters representing lesion texture, the image filters areapplied to lesions as represented in the training data. For imagefilters representing healthy tissue or a combination of healthy tissueand lesion tissue, the image filters are applied to the regions ofinterest for that combination.

The image filters are applied by spatial filtering. The image filtersare one, two, or three-dimensional filters. The N filters are applied toeach tumor by a convolution operation (e.g., dot product or othersimilarity measure). The image filter is positioned relative to thelegion and the intensities of the image filter are multiplied with thedata of the image in the patch of the region of interest. A sum oraverage of the resulting multiplications provides a value for the imagefilter at that position. By shifting the image filter, such as was donefor the patches, values for different locations are determined. Anyconvolution of the image filters with the lesions as represented in thetraining frames of data may be used.

FIGS. 5A-C show convolution results of the image filters of FIG. 4 withthree different lesions. The left to right sequence of texture featuresrepresented by the image filters of FIG. 4 is maintained in FIGS. 5A-C.The left most CT slice image is of the lesion being filtered. Where thelesion includes texture more closely resembling the image filter, theresulting pixel has a higher intensity (e.g., more white). Thedifferences in scale between FIGS. 5A-C are due to differences in scaleand/or size of the lesion.

The resulting values (i.e., filter responses) from the convolution arecombined to indicate a feature value for that lesion based on the givenimage filter. For example, the intensity values are summed or averaged.In the example of FIG. 5A, the average intensity for each of the texturefeatures is calculated, providing a single value for each image filter.For example, the average response per filter per tumor is recorded as avalue of the ISA feature. Alternatively, a feature is calculated fromthe results of the convolution, such as using a Haar wavelet feature orother pre-designed feature from the data driven feature provided by theimage filter convolution with the image data.

The resulting values are the texture features as used for the trainingin act 52. Other information may be used as part of the input featurevector for training. The other information may represent the patient(e.g., age, sex, or family history), stage of the lesion, type oftherapy, other situational information, and/or other image derivedfeatures.

In one embodiment, other texture features are determined from thetraining data. For example, Haralick texture features, such as contrast,entropy, and/or others, are used. In another example or in addition,homogeneity, energy, and dissimilarity texture features are used. Thehomogeneity, for example, has the following equation:

HGT=Σ _(i=1) ^(N) ^(g) Σ_(j=1) ^(N) ^(g) P(i,j)/(1+|i−j|)

where P^((i, j)) counts the number of times a pixel with value i isadjacent to a pixel with value j and then dividing by the total numberof such comparisons made. N_(g) is the number of gray levels in theimage. The texture features rely on predetermined formulas rather thanfeatures learned from training data by a processor. Any now known orlater developed texture features may be used.

In another additional or alternative embodiment, texture features basedon local binary patterns are used. The local binary patterns compare theintensity of each pixel with its neighboring pixels and return a codethat compactly summarizes the differences. These codes are thensummarized over the ROI through a histogram. A data pyramid may be used,where the same image at different resolutions (e.g., by decimation) isprovided. The local binary patterns are then calculated for eachresolution or level of the pyramid, as each pyramid level containstextural information in different scale and details.

In one embodiment, the texture features are calculated from the trainingdata for one pre-therapy scan and one post therapy scan. In alternativeembodiments, only pre-therapy scan, only the first post therapy scan, oradditional post therapy scans are used.

The texture features, whether data driven or other texture features, arethe same for pre and post therapy scans. In alternative embodiments,different features are provide for different scans relative to thetherapy (i.e., learn texture features for pre-therapy images differentthan texture features learned from post therapy images).

Once the values for the input feature vector are calculated and/orobtained, the ground truth for therapy response is used to train, by aprocessor, the predictor of the therapy response. The ground truth isbinary, such as successful or not successful outcome from therapy.Alternatively, the ground truth has a greater resolution, such as thenumber of months without re-occurrence, time of remission, change insize, time to narcotization, or other measure of success of therapy.

In act 52, a predictor of therapy response is learned by a processor.The processor uses the ground truth and data-driven texture features,with or without other features to train a classifier. Any machinelearning may be used, such as a probabilistic boosting tree, supportvector machine, or other machine learning classifier. Other classifiersmay include a single class or binary classifier, collection of differentclassifiers, cascaded classifiers, hierarchal classifier, multi-classclassifier, model-based classifier, or combinations thereof may be used.Multi-class classifiers include CART, K-nearest neighbors, neuralnetwork (e.g., multi-layer perceptron), mixture models, or others.Error-correcting output code (ECOC) may be used.

In one embodiment, the features are fed into a machine learningclassifier or regressor, such as a Support Vector Machine (SVM) orRegression Random Forest, for training. The training provides a matrixassociating input values of features to outputs or predictions. Thematrix or trained classifier (e.g., learnt classifier) is a predictor ofthe therapy response. During training, the set of ground truth caseswith known therapy outcome are provided for learning the systemparameters. The predictor may be used predict the therapy outcome earlyin the therapy process, such as from one, two, or less than five scans.

Different predictors may be learnt using different training data. Thesystem may be trained for different lesion entities (liver, lung, lymphnodes, or other), different therapies, different cohorts of patients,different imaging modality, or other differences, resulting in specifictherapy response features that are automatically learnt. Either separatesystems can be trained or the system may incorporate additionalnon-image features, such as the type of therapy, lesion entity, or timeafter onset of therapy to augment image-based features. Alternatively, apredictor generic to different situations (e.g., different types oflesions, different organs, or different imaging modality) is trainedfrom training data with the same generic situation.

During the optimization to train the predictor, different distinguishingfeatures are learned. Less than all of the input features may bedeterminative in a given situation. The training may select somefeatures and not others to be used for predicting therapy response. Thelearning may indicate only a sub-set of features to be used forclassification. In alternative embodiments, all of the features of theinput feature vector are used by the learnt predictor.

The learnt predictor is applied to assess therapy response for a givenpatient. For patients with a same type of lesion or other situation, thesame learnt predictor may be used. FIG. 6 shows one embodiment ofapplication for therapy response assessment. The same or a differentprocessor than used for training performs the acts of FIG. 6. Forexample, the matrix is used by a server for online assessment bycustomers. As another example, a medical institution purchases thematrix and applies the matrix for use with their patients.

In act 54, one or more images are obtained. The images are obtained bytransfer, loading from memory, or as output by scanning. The images arepre and/or post therapy images for the patient. In one embodiment, theimages are from a pre-therapy scan and only a few (e.g., only one oronly two) post therapy scans. In yet other embodiments, an image orimages from only one scan (e.g., pre or post therapy) are used.

The images represent a lesion in the patient. Multiple lesions may berepresented. The data representing the lesion is obtained by medicalimaging, such as from CT scanning with a CT system. An x-ray source anddetector mounted on a gantry scan the patient. A computer reconstructs avolume of the patient from the detected x-ray attenuations. A slice orprojection from the volume may be used as a frame of data of an imagerepresenting the lesion. Other computed tomography images may be used,such as contrast enhanced or iodine images.

In act 56, one or more regions of interest are identified. The lesion asrepresented in each of the images is located. The same or differentsegmentation or region designation used for training is used inapplication. For example, a processor automatically segments the legionfrom healthy tissue. As another example, a user manually positions a boxdesignating the region of interest.

The region of interest includes the lesion and a border of the lesion.Surrounding information may or may not be included. The region may ormay not be dilated to include the surrounding information.

In act 58, texture features are extracted with independent analysis. Theindependent analysis was performed for learning the texture featuresfrom the training data. The resulting texture feature is used in theapplication to extract one or more values. The image filters or filterkernels learned from the training data are convolved with the lesions asrepresented by the obtained images. The filter kernels are convolvedwith the regions of interest.

The convolution is performed for each of the images representing thelesion. The pre and/or post therapy medical images, at least for theregions of interest, are convolved with the filter kernels. Any numberof filter kernels is used, such as three or more (e.g., five in theexample of FIG. 4). Each filter kernel is convolved with each of theregions of interest in each of the images.

Since only one, only two, or a few number of scans are performed forearly prediction, the texture features are extracted from this limitednumber of images. The filter kernels used to extract the texturefeatures are learned from the images of many patients, but the resultingfilter kernels are applied to the images of a given patient forapplication.

The texture feature (e.g., filter kernel) to be convolved with the imageis learnt automatically from training images with or without groundtruth information. The ground truth may be used to identify texture fordesired outcome from texture with undesired outcome or therapy response.The texture feature for convolution with the images is based onindependent analysis, such as independent subspace analysis. By usingdata to develop the texture feature, different texture features resultfrom the use of different training data. Training data appropriate for agiven application is used to provide the texture feature.

The convolution is limited to the appropriate regions of interest. Forexample, filter kernels developed for lesions are used to filter regionsof interest corresponding to or representing lesions. The filtering isnot performed outside of the region of interest.

The filter kernels are used to filter the images of the patient. Thesame or different calculation used to calculate feature value fortraining the classifier are used to calculate the value for the texturefeature. For example, the results of the convolution are summed oraveraged. The intensities output from the filtering are summed. The summay be divided by the number of samples (e.g., pixels) in the region ofinterest.

In act 60, other features are obtained. Other texture features may beextracted. For example, filters defined by mathematical formulas areconvolved with the images. Any formula-based texture features may beused, such as discussed above for training. Pre-designed texturefeatures, such as Haar wavelets, homogeneity, or local binary patterns,may be calculated.

Other non-texture features may be calculated. The features may begeometric, such as change in area or volume. The features may be patientrelated, such as smoking history, medications, or family history. Thefeatures may be a stage or other score related to the lesion.

In act 62, a therapy response of the lesion of the patient isclassified. A processor, implementing a machine-learnt classifier,predicts the therapy response. The features, such as one or morefeatures resulting from convolving the data-driven or trainingdata-based texture feature, are input to the predictor. The predictedresponse to therapy based on the input feature vector is output. Theoutcome of therapy is predicted as a function of the texture features.

In one embodiment, the prediction is performed without any geometricmeasures as input. In other embodiments, an area or volume may beincluded in the feature vector, but shrinkage or change over time ofgeometry is not included. Instead, texture information is used. In otherembodiments, shrinkage or other change in geometry over time is used asan input feature with texture features.

In one embodiment, the therapy response is classified with just imagingfeatures from data-driven or training data learnt texture features. Inother embodiments, other imaging features are used, such as texturefeatures from pre-designed or mathematical formula-based texturefeatures in the classifier input feature vector.

For features from imaging, only features from the pre-therapy medicalimages and a limited number of post therapy medical images are used. Thelimited number of post therapy medical images corresponds to only onescan, only two scans, or only three scans. This provides for earlyassessment of therapy response. Additional images from other scans maybe used in alternative embodiments. In an alternative embodiment, onlyone image, or images from only one scan are used, such as predictingtherapy response from pre-therapy images or from a single post therapyscan.

The classification is performed with any machine-learnt classifier. Forexample, the predictor is a support vector machine or a regressionrandom forest. Other classifiers may be used.

The classifier predicts an outcome of a given therapy. The predictionmay occur after one round of therapy, so the classifier predicts anoutcome of continuing therapy. After obtaining a post therapy scan andcorresponding medical image or images, the success or not of the therapysequence is predicted before progressing further in the therapysequence. Alternatively, the outcome is predicted after therapy iscomplete, but before the lesion fully responds to the therapy.

The classifier outputs a likelihood of therapy success or failure forthe lesion of the patient. The likelihood is a binary indication, suchas success or failure (e.g., lesion eradicated or not, or lesion growthstopped or not). Alternatively, the likelihood is a success rating witha scale of three or more. For example, the likelihood is a percentagechance of success or failure of treatment. A SVM, Regression RandomForest, Bayesian, or other machine-learnt classifier may provide aprobability as an output. The probability is a ranking or percentagelikelihood of success or failure in treatment of the lesion.

The classifier may be trained based on any definition of success orfailure. For example, stopping growth, causing a percentage shrinkage,eradicating, or causing necrosis may be used as the definition ofsuccess.

In act 64, the likelihood of therapy response is output. The output isto a display, to a computer, or to storage. For example, the likelihoodis added to a computerized medical record for the patient. As anotherexample, an image or multiple images of the lesion are displayed to theuser. The likelihood is indicated on or adjacent to the image. Colorcoding, text, or a numerical value may be used to represent thelikelihood.

FIG. 7 shows a system for therapy response assessment. The system is ahost computer, control station, work station, server, or otherarrangement. The system includes the display 14, memory 16, andprocessor 18. Additional, different, or fewer components may beprovided. The system is for training, such as using images from themedical imaging system 11 as ground truth. Alternatively, the system isfor application of the learned features and classifier, such as usingimages from the medical imaging system 11 for predicting response totherapy for a patient. In other embodiments, the medical imaging system11 is part of the system. In yet other embodiments, a picture archivingand communications system (PACS) or other memory is provided instead ofor in addition to the medical imaging system 11 for supplying images.

The display 14, processor 18, and memory 16 may be part of a computer,server, or other system for image processing images from the medicalimaging system 11. A workstation or control station for the medicalimaging system 11 may be used. Alternatively, a separate or remotedevice not part of the medical imaging system 11 is used. Instead, thetherapy assessment is performed remotely from the medical imaging system11.

In one embodiment, the processor 18 and memory 16 are part of a serverhosting the therapy response assessment function for use by a medicalprofessional computer as the client. The client and server areinterconnected by a network, such as an intranet or the Internet. Theclient may be a computer of the medical imaging system 11 or a computerof a medical professional, and the server may be provided by amanufacturer, provider, host, or creator of the therapy responseassessment system.

The medical imaging system 11 is any now known or later developedimaging system. For example, the medical imaging system 11 is a computedtomography, ultrasound, x-ray, magnetic resonance, or functional imagingsystem. As a computed tomography system, an x-ray source and detectorare mounted on or in a gantry on opposite sides of a patient space andcorresponding patient bed. As the gantry moves the source and detectoraround the patient, a sequence of x-ray projections of the patient areacquired. A processor, such as the processor 18 or a differentprocessor, reconstructs the x-ray attenuation in three-dimensions or forone or more slices.

The display 14 is a CRT, LCD, projector, plasma, printer, smart phone orother now known or later developed display device for displaying theimages, learned texture features, and/or therapy response assessmentinformation. For example, the display 14 displays two images,information about the images, therapy information, and an indication ofwhether the therapy is predicted to be successful. In a trainingenvironment, the display 14 may be of data-driven features, statistics,feature information, optimization information, or other traininginformation.

The texture features (e.g., data learnt texture features), otherfeatures, classifiers, matrices, outputs, images, regions of interest,patches, and/or other information are stored in a non-transitorycomputer readable memory, such as the memory 16. The memory 16 is anexternal storage device, RAM, ROM, database, and/or a local memory(e.g., solid state drive or hard drive). The same or differentnon-transitory computer readable media may be used for instructions andother data. The memory 16 may be implemented using a database managementsystem (DBMS) managed by the processor 18 and residing on a memory, suchas a hard disk, RAM, or removable media. Alternatively, the memory 16 isinternal to the processor 18 (e.g. cache).

The instructions for implementing the therapy response assessment intraining or application processes, methods and/or techniques discussedherein are provided on non-transitory computer-readable storage media ormemories, such as a cache, buffer, RAM, removable media, hard drive orother computer readable storage media (e.g., the memory 16). Computerreadable storage media include various types of volatile and nonvolatilestorage media. The functions, acts or tasks illustrated in the figuresor described herein are executed in response to one or more sets ofinstructions stored in or on computer readable storage media. Thefunctions, acts or tasks are independent of the particular type ofinstructions set, storage media, processor or processing strategy andmay be performed by software, hardware, integrated circuits, firmware,micro code and the like, operating alone or in combination.

In one embodiment, the instructions are stored on a removable mediadevice for reading by local or remote systems. In other embodiments, theinstructions are stored in a remote location for transfer through acomputer network. In yet other embodiments, the instructions are storedwithin a given computer, CPU, GPU or system. Because some of theconstituent system components and method steps depicted in theaccompanying figures may be implemented in software, the actualconnections between the system components (or the process steps) maydiffer depending upon the manner in which the present embodiments areprogrammed.

A program may be uploaded to, and executed by, the processor 18comprising any suitable architecture. Likewise, processing strategiesmay include multiprocessing, multitasking, parallel processing and thelike. The processor 18 is implemented on a computer platform havinghardware, such as one or more central processing units (CPU), a randomaccess memory (RAM), and input/output (I/O) interface(s). The computerplatform also includes an operating system and microinstruction code.The various processes and functions described herein may be either partof the microinstruction code or part of the program (or combinationthereof) which is executed via the operating system. Alternatively, theprocessor 18 is one or more processors in a network.

The processor 18 is configured to obtain images. A region of interestmay be determined by the processor 18 or received from a user interface.

The processor 18 is configured to learn features or extract learnedfeatures. For example, an independent analysis is performed on acollection of regions of interest representing lesions. As anotherexample, a learnt texture feature, such as a filter kernel fromindependent analysis, is applied by the processor 18. The processor 18may train a classifier using the learned or data-driven features,training data, and ground truth information.

The processor 18 is configured to classify based on the learnedfeatures. The features are extracted from one or more images for a givenpatient. The values of the features are input to a learnt classifier.The processor 18 determines an output of the machine-learnt classifierbased on the input features. The output is provided to the memory 16,the display 14, or a network interface.

Various improvements described herein may be used together orseparately. Although illustrative embodiments of the present inventionhave been described herein with reference to the accompanying drawings,it is to be understood that the invention is not limited to thoseprecise embodiments, and that various other changes and modificationsmay be affected therein by one skilled in the art without departing fromthe scope or spirit of the invention.

What is claimed is:
 1. A method for therapy response assessment, themethod comprising: obtaining a pre-therapy medical image of a patient,the pre-therapy medical image representing at least one lesion of thepatient; obtaining a post-therapy medical image of the patient, thepost-therapy medical image of the patient representing the at least onelesion of the patient; convolving, by a processor, the pre-therapy andpost therapy medical images with a texture feature learned from trainingimages; and classifying, by the processor, a therapy response of thelesion with a machine-learnt classifier with a result of the convolvingas an input feature to the machine-learnt classifier.
 2. The method ofclaim 1 wherein obtaining the pre-therapy and post therapy medicalimages comprises obtaining computed tomography images.
 3. The method ofclaim 1 wherein obtaining the post-therapy medical image comprisesobtaining only the post-therapy medical image or only the post-therapymedical image and one more post therapy medical image, and whereinclassifying comprises classifying with the input feature including, forjust features from imaging, only features from the pre-therapy andpost-therapy medical images.
 4. The method of claim 1 wherein convolvingcomprises convolving with the texture feature comprising an independentsub-space analysis feature.
 5. The method of claim 1 wherein convolvingcomprises convolving with the texture feature and at least two othertexture features learned from the training images.
 6. The method ofclaim 1 wherein convolving comprises convolving with the texture featurelearned automatically from the training images and labeled groundtruths.
 7. The method of claim 1 wherein convolving comprises filteringwith a kernel defined by the texture feature and summing intensitiesoutput from the filtering, the result being a function of the sum of theintensities.
 8. The method of claim 1 wherein convolving comprisesconvolving with the texture feature comprising a training image-basedfeature such that different training images result in different texturefeatures.
 9. The method of claim 1 wherein classifying comprisesclassifying with a support vector machine or a regression random forest.10. The method of claim 1 wherein classifying comprises predicting anoutcome of continuing therapy after obtaining the post-therapy medicalimage.
 11. The method of claim 1 wherein classifying comprisesindicating a likelihood of therapy success or failure for the lesion.12. The method of claim 1 further comprising identifying regions ofinterest in the pre-therapy and post-therapy medical images includingthe lesion and a border of the lesion, and wherein convolving comprisesconvolving the texture feature with the regions of interest and not withregions outside the regions of interest.
 13. The method of claim 1further comprising: convolving, by the processor, the pre-therapy andpost therapy medical images with mathematical formula-based texturefeatures; wherein classifying comprises classifying the therapy responsewith the result and results from the convolving with the mathematicalformula-based textures features as an input vector including the inputfeature.
 14. In a non-transitory computer readable storage medium havingstored therein data representing instructions executable by a programmedprocessor for therapy response assessment, the storage medium comprisinginstructions for: with only one or two scans, extracting texturefeatures for a lesion with a filter kernel, the filter kernel beingindependently based on image data; and predicting an outcome of therapyon the lesion, the predicting being a function of the texture features.15. The non-transitory computer readable storage medium of claim 14wherein extracting the texture features comprises convolving scan dataof the one or two scans with the filter kernel and additional filterkernels, the filter kernel and additional filter kernels being developedfrom independent subspace analysis of the image data, the image datacomprising training images.
 16. The non-transitory computer readablestorage medium of claim 14 wherein predicting comprises predicting witha machine trained classifier without consideration of shrinkage of thelesion.
 17. The non-transitory computer readable storage medium of claim14 further comprising outputting a likelihood of the outcome to adisplay.
 18. A method for therapy response assessment, the methodcomprising: subjecting, by a processor, patches of lesions representedin a plurality of training frames of data to independent subspaceanalysis; creating, by the processor, a number of image filters from theindependent subspace analysis of the patches; calculating, by theprocessor, texture features by application of the image filters to thelesions as represented in the training frames of data; and learning, bythe processor, a predictor of therapy response as a function of thetexture features.
 19. The method of claim 18 further comprisingwhitening the patches with principle component analysis prior to thecreating.
 20. The method of claim 18 wherein calculating the texturefeatures comprises convolving the image filters with the lesions asrepresented in the training frames of data and averaging responses ofthe convolving, the averages comprising the texture features.